

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Cephadm Operations &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/doctools.js"></script>
        <script src="../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="Basic Ceph Client Setup" href="../client-setup/" />
    <link rel="prev" title="升级 Ceph" href="../upgrade/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../">Cephadm</a></li>
      <li class="breadcrumb-item active">Cephadm Operations</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../_sources/cephadm/operations.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../">Cephadm</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../compatibility/">Compatibility and Stability</a></li>
<li class="toctree-l2"><a class="reference internal" href="../install/">部署个全新的 Ceph 集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../adoption/">现有集群切换到 cephadm</a></li>
<li class="toctree-l2"><a class="reference internal" href="../host-management/">Host Management</a></li>
<li class="toctree-l2"><a class="reference internal" href="../services/">Service Management</a></li>
<li class="toctree-l2"><a class="reference internal" href="../certmgr/">Certificate Management</a></li>
<li class="toctree-l2"><a class="reference internal" href="../upgrade/">升级 Ceph</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Cephadm operations</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#watching-cephadm-log-messages">Watching cephadm log messages</a></li>
<li class="toctree-l3"><a class="reference internal" href="#ceph-daemon-control">Ceph daemon control</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#starting-and-stopping-daemons">Starting and stopping daemons</a></li>
<li class="toctree-l4"><a class="reference internal" href="#redeploying-or-reconfiguring-a-daemon">Redeploying or reconfiguring a daemon</a></li>
<li class="toctree-l4"><a class="reference internal" href="#rotating-a-daemon-s-authenticate-key">Rotating a daemon’s authenticate key</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#ceph-daemon-logs">Ceph daemon logs</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#logging-to-journald">Logging to journald</a></li>
<li class="toctree-l4"><a class="reference internal" href="#logging-to-files">Logging to files</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#per-node-cephadm-logs">Per-node cephadm logs</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#setting-a-cephadm-log-destination-during-bootstrap">Setting a cephadm log destination during bootstrap</a></li>
<li class="toctree-l4"><a class="reference internal" href="#setting-a-cephadm-log-destination-on-an-existing-cluster">Setting a cephadm log destination on an existing cluster</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#data-location">Data location</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#disk-usage">Disk usage</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#health-checks">Health checks</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#id1">CEPHADM Operations</a></li>
<li class="toctree-l4"><a class="reference internal" href="#cluster-configuration-checks">Cluster Configuration Checks</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#client-keyrings-and-configs">Client keyrings and configs</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#listing-client-keyrings">Listing Client Keyrings</a></li>
<li class="toctree-l4"><a class="reference internal" href="#putting-a-keyring-under-management">Putting a Keyring Under Management</a></li>
<li class="toctree-l4"><a class="reference internal" href="#disabling-management-of-a-keyring-file">Disabling Management of a Keyring File</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#etc-ceph-ceph-conf">/etc/ceph/ceph.conf</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#distributing-ceph-conf-to-hosts-that-have-no-keyrings">Distributing ceph.conf to hosts that have no keyrings</a></li>
<li class="toctree-l4"><a class="reference internal" href="#using-placement-specs-to-specify-which-hosts-get-keyrings">Using Placement Specs to specify which hosts get keyrings</a></li>
<li class="toctree-l4"><a class="reference internal" href="#distributing-ceph-conf-to-hosts-tagged-with-bare-config">Distributing ceph.conf to hosts tagged with bare_config</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#limiting-password-less-sudo-access">Limiting Password-less sudo Access</a></li>
<li class="toctree-l3"><a class="reference internal" href="#purging-a-cluster">Purging a cluster</a></li>
<li class="toctree-l3"><a class="reference internal" href="#replacing-a-device">Replacing a device</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../client-setup/">Client Setup</a></li>
<li class="toctree-l2"><a class="reference internal" href="../troubleshooting/">Troubleshooting</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dev/cephadm/">Cephadm Feature Planning</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="cephadm-operations">
<h1>Cephadm Operations<a class="headerlink" href="#cephadm-operations" title="Permalink to this heading"></a></h1>
<section id="watching-cephadm-log-messages">
<span id="watching-cephadm-logs"></span><h2>Watching cephadm log messages<a class="headerlink" href="#watching-cephadm-log-messages" title="Permalink to this heading"></a></h2>
<p>The cephadm orchestrator module writes logs to the <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> cluster log
channel. You can monitor Ceph’s activity in real time by reading the logs as
they fill up. Run the following command to see the logs in real time:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "# ";
}
</style><span class="prompt1">ceph<span class="w"> </span>-W<span class="w"> </span>cephadm</span>
</pre></div></div><p>By default, this command shows info-level events and above.  To see
debug-level messages as well as info-level events, run the following
commands:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/log_to_cluster_level<span class="w"> </span>debug</span>
<span class="prompt1">ceph<span class="w"> </span>-W<span class="w"> </span>cephadm<span class="w"> </span>--watch-debug</span>
</pre></div></div><div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>The debug messages are very verbose!</p>
</div>
<p>You can see recent events by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>log<span class="w"> </span>last<span class="w"> </span>cephadm</span>
</pre></div></div><p>These events are also logged to the <code class="docutils literal notranslate"><span class="pre">ceph.cephadm.log</span></code> file on
monitor hosts as well as to the monitor daemons’ stderr.</p>
</section>
<section id="ceph-daemon-control">
<span id="cephadm-logs"></span><h2>Ceph daemon control<a class="headerlink" href="#ceph-daemon-control" title="Permalink to this heading"></a></h2>
<section id="starting-and-stopping-daemons">
<h3>Starting and stopping daemons<a class="headerlink" href="#starting-and-stopping-daemons" title="Permalink to this heading"></a></h3>
<p>You can stop, start, or restart a daemon with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>stop<span class="w"> </span>&lt;name&gt;</span>
<span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>start<span class="w"> </span>&lt;name&gt;</span>
<span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>restart<span class="w"> </span>&lt;name&gt;</span>
</pre></div></div><p>You can also do the same for all daemons for a service with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>stop<span class="w"> </span>&lt;name&gt;</span>
<span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>start<span class="w"> </span>&lt;name&gt;</span>
<span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>restart<span class="w"> </span>&lt;name&gt;</span>
</pre></div></div></section>
<section id="redeploying-or-reconfiguring-a-daemon">
<h3>Redeploying or reconfiguring a daemon<a class="headerlink" href="#redeploying-or-reconfiguring-a-daemon" title="Permalink to this heading"></a></h3>
<p>The container for a daemon can be stopped, recreated, and restarted with
the <code class="docutils literal notranslate"><span class="pre">redeploy</span></code> command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>redeploy<span class="w"> </span>&lt;name&gt;<span class="w"> </span><span class="o">[</span>--image<span class="w"> </span>&lt;image&gt;<span class="o">]</span></span>
</pre></div></div><p>A container image name can optionally be provided to force a
particular image to be used (instead of the image specified by the
<code class="docutils literal notranslate"><span class="pre">container_image</span></code> config value).</p>
<p>If only the ceph configuration needs to be regenerated, you can also
issue a <code class="docutils literal notranslate"><span class="pre">reconfig</span></code> command, which will rewrite the <code class="docutils literal notranslate"><span class="pre">ceph.conf</span></code>
file but will not trigger a restart of the daemon.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>reconfig<span class="w"> </span>&lt;name&gt;</span>
</pre></div></div></section>
<section id="rotating-a-daemon-s-authenticate-key">
<h3>Rotating a daemon’s authenticate key<a class="headerlink" href="#rotating-a-daemon-s-authenticate-key" title="Permalink to this heading"></a></h3>
<p>All Ceph and gateway daemons in the cluster have a secret key that is used to connect
to and authenticate with the cluster.  This key can be rotated (i.e., replaced with a
new key) with the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>rotate-key<span class="w"> </span>&lt;name&gt;</span>
</pre></div></div><p>For MDS, OSD, and MGR daemons, this does not require a daemon restart.  For other
daemons, however (e.g., RGW), the daemon may be restarted to switch to the new key.</p>
</section>
</section>
<section id="ceph-daemon-logs">
<h2>Ceph daemon logs<a class="headerlink" href="#ceph-daemon-logs" title="Permalink to this heading"></a></h2>
<section id="logging-to-journald">
<h3>Logging to journald<a class="headerlink" href="#logging-to-journald" title="Permalink to this heading"></a></h3>
<p>Ceph daemons traditionally write logs to <code class="docutils literal notranslate"><span class="pre">/var/log/ceph</span></code>. Ceph daemons log to
journald by default and Ceph logs are captured by the container runtime
environment. They are accessible via <code class="docutils literal notranslate"><span class="pre">journalctl</span></code>.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Prior to Quincy, ceph daemons logged to stderr.</p>
</div>
<section id="example-of-logging-to-journald">
<h4>Example of logging to journald<a class="headerlink" href="#example-of-logging-to-journald" title="Permalink to this heading"></a></h4>
<p>For example, to view the logs for the daemon <code class="docutils literal notranslate"><span class="pre">mon.foo</span></code> for a cluster
with ID <code class="docutils literal notranslate"><span class="pre">5c5a50ae-272a-455d-99e9-32c6a013e694</span></code>, the command would be
something like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">journalctl<span class="w"> </span>-u<span class="w"> </span>ceph-5c5a50ae-272a-455d-99e9-32c6a013e694@mon.foo</span>
</pre></div></div><p>This works well for normal operations when logging levels are low.</p>
</section>
</section>
<section id="logging-to-files">
<h3>Logging to files<a class="headerlink" href="#logging-to-files" title="Permalink to this heading"></a></h3>
<p>You can also configure Ceph daemons to log to files instead of to
journald if you prefer logs to appear in files (as they did in earlier,
pre-cephadm, pre-Octopus versions of Ceph).  When Ceph logs to files,
the logs appear in <code class="docutils literal notranslate"><span class="pre">/var/log/ceph/&lt;cluster-fsid&gt;</span></code>. If you choose to
configure Ceph to log to files instead of to journald, remember to
configure Ceph so that it will not log to journald (the commands for
this are covered below).</p>
<section id="enabling-logging-to-files">
<h4>Enabling logging to files<a class="headerlink" href="#enabling-logging-to-files" title="Permalink to this heading"></a></h4>
<p>To enable logging to files, run the following commands:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>global<span class="w"> </span>log_to_file<span class="w"> </span><span class="nb">true</span></span>
<span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>global<span class="w"> </span>mon_cluster_log_to_file<span class="w"> </span><span class="nb">true</span></span>
</pre></div></div></section>
<section id="disabling-logging-to-journald">
<h4>Disabling logging to journald<a class="headerlink" href="#disabling-logging-to-journald" title="Permalink to this heading"></a></h4>
<p>If you choose to log to files, we recommend disabling logging to journald or else
everything will be logged twice. Run the following commands to disable logging
to stderr:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>global<span class="w"> </span>log_to_stderr<span class="w"> </span><span class="nb">false</span></span>
<span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>global<span class="w"> </span>mon_cluster_log_to_stderr<span class="w"> </span><span class="nb">false</span></span>
<span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>global<span class="w"> </span>log_to_journald<span class="w"> </span><span class="nb">false</span></span>
<span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>global<span class="w"> </span>mon_cluster_log_to_journald<span class="w"> </span><span class="nb">false</span></span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>You can change the default by passing --log-to-file during
bootstrapping a new cluster.</p>
</div>
</section>
<section id="modifying-the-log-retention-schedule">
<h4>Modifying the log retention schedule<a class="headerlink" href="#modifying-the-log-retention-schedule" title="Permalink to this heading"></a></h4>
<p>By default, cephadm sets up log rotation on each host to rotate these
files.  You can configure the logging retention schedule by modifying
<code class="docutils literal notranslate"><span class="pre">/etc/logrotate.d/ceph.&lt;cluster-fsid&gt;</span></code>.</p>
</section>
</section>
</section>
<section id="per-node-cephadm-logs">
<h2>Per-node cephadm logs<a class="headerlink" href="#per-node-cephadm-logs" title="Permalink to this heading"></a></h2>
<p>The cephadm executable, either run directly by a user or by the cephadm
orchestration module, may also generate logs. It does so independently of
the other Ceph components running in containers. By default, this executable
logs to the file <code class="docutils literal notranslate"><span class="pre">/var/log/ceph/cephadm.log</span></code>.</p>
<p>This logging destination is configurable and you may choose to log to the
file, to the syslog/journal, or to both.</p>
<section id="setting-a-cephadm-log-destination-during-bootstrap">
<h3>Setting a cephadm log destination during bootstrap<a class="headerlink" href="#setting-a-cephadm-log-destination-during-bootstrap" title="Permalink to this heading"></a></h3>
<p>The <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> command may be executed with the option <code class="docutils literal notranslate"><span class="pre">--log-dest=file</span></code>
or with <code class="docutils literal notranslate"><span class="pre">--log-dest=syslog</span></code> or both. These options control where cephadm
will store persistent logs for each invocation. When these options are
specified for the <code class="docutils literal notranslate"><span class="pre">cephadm</span> <span class="pre">bootstrap</span></code> command the system will automatically
record these settings for future invocations of <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> by the cephadm
orchestration module.</p>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">cephadm<span class="w"> </span>--log-dest<span class="o">=</span>syslog<span class="w"> </span>bootstrap<span class="w"> </span><span class="c1"># ... other bootstrap arguments ...</span></span>
</pre></div></div><p>If you want to manually specify exactly what log destination to use
during bootstrap, independent from the <code class="docutils literal notranslate"><span class="pre">--log-dest</span></code> options, you may add
a configuration key <code class="docutils literal notranslate"><span class="pre">mgr/cephadm/cephadm_log_destination</span></code> to the
initial configuration file, under the <code class="docutils literal notranslate"><span class="pre">[mgr]</span></code> section. Valid values for
the key are: <code class="docutils literal notranslate"><span class="pre">file</span></code>, <code class="docutils literal notranslate"><span class="pre">syslog</span></code>, and <code class="docutils literal notranslate"><span class="pre">file,syslog</span></code>.</p>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">cat<span class="w"> </span>&gt;/tmp/bootstrap.conf<span class="w"> </span>&lt;&lt;EOF</span>
<span class="prompt1"><span class="o">[</span>mgr<span class="o">]</span></span>
<span class="prompt1">mgr/cephadm/cephadm_log_destination<span class="w"> </span><span class="o">=</span><span class="w"> </span>syslog</span>
<span class="prompt1">EOF</span>
<span class="prompt1">cephadm<span class="w"> </span>bootstrap<span class="w"> </span>--config<span class="w"> </span>/tmp/bootstrap.conf<span class="w"> </span><span class="c1"># ... other bootstrap arguments ...</span></span>
</pre></div></div></section>
<section id="setting-a-cephadm-log-destination-on-an-existing-cluster">
<h3>Setting a cephadm log destination on an existing cluster<a class="headerlink" href="#setting-a-cephadm-log-destination-on-an-existing-cluster" title="Permalink to this heading"></a></h3>
<p>An existing Ceph cluster can be configured to use a specific cephadm log
destination by setting the <code class="docutils literal notranslate"><span class="pre">mgr/cephadm/cephadm_log_destination</span></code>
configuration value to one of <code class="docutils literal notranslate"><span class="pre">file</span></code>, <code class="docutils literal notranslate"><span class="pre">syslog</span></code>, or <code class="docutils literal notranslate"><span class="pre">file,syslog</span></code>. This
will cause the cephadm orchestration module to run <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> so that logs go
to <code class="docutils literal notranslate"><span class="pre">/var/log/ceph/cephadm.log</span></code>, the syslog/journal, or both, respectively.</p>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="c1"># set the cephadm executable to log to syslog</span></span>
<span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/cephadm_log_destination<span class="w"> </span>syslog</span>
<span class="prompt1"><span class="c1"># set the cephadm executable to log to both the log file and syslog</span></span>
<span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/cephadm_log_destination<span class="w"> </span>file,syslog</span>
<span class="prompt1"><span class="c1"># set the cephadm executable to log to the log file</span></span>
<span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/cephadm_log_destination<span class="w"> </span>file</span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>If you execute cephadm commands directly, such as cephadm shell,
this option will not apply. To have cephadm log to locations other than
the default log file When running cephadm commands directly use the
<code class="docutils literal notranslate"><span class="pre">--log-dest</span></code> options described in the bootstrap section above.</p>
</div>
</section>
</section>
<section id="data-location">
<h2>Data location<a class="headerlink" href="#data-location" title="Permalink to this heading"></a></h2>
<p>Cephadm stores daemon data and logs in different locations than did
older, pre-cephadm (pre Octopus) versions of ceph:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">/var/log/ceph/&lt;cluster-fsid&gt;</span></code> contains all cluster logs. By
default, cephadm logs via stderr and the container runtime. These
logs will not exist unless you have enabled logging to files as
described in <a class="reference internal" href="#cephadm-logs">cephadm-logs</a>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">/var/lib/ceph/&lt;cluster-fsid&gt;</span></code> contains all cluster daemon data
(besides logs).</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">/var/lib/ceph/&lt;cluster-fsid&gt;/&lt;daemon-name&gt;</span></code> contains all data for
an individual daemon.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">/var/lib/ceph/&lt;cluster-fsid&gt;/crash</span></code> contains crash reports for
the cluster.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">/var/lib/ceph/&lt;cluster-fsid&gt;/removed</span></code> contains old daemon
data directories for stateful daemons (e.g., monitor, prometheus)
that have been removed by cephadm.</p></li>
</ul>
<section id="disk-usage">
<h3>Disk usage<a class="headerlink" href="#disk-usage" title="Permalink to this heading"></a></h3>
<p>Because a few Ceph daemons (notably, the monitors and prometheus) store a
large amount of data in <code class="docutils literal notranslate"><span class="pre">/var/lib/ceph</span></code> , we recommend moving this
directory to its own disk, partition, or logical volume so that it does not
fill up the root file system.</p>
</section>
</section>
<section id="health-checks">
<h2>Health checks<a class="headerlink" href="#health-checks" title="Permalink to this heading"></a></h2>
<p>The cephadm module provides additional health checks to supplement the
default health checks provided by the Cluster. These additional health
checks fall into two categories:</p>
<ul class="simple">
<li><p><strong>cephadm operations</strong>: Health checks in this category are always
executed when the cephadm module is active.</p></li>
<li><p><strong>cluster configuration</strong>: These health checks are <em>optional</em>, and
focus on the configuration of the hosts in the cluster.</p></li>
</ul>
<section id="id1">
<h3>CEPHADM Operations<a class="headerlink" href="#id1" title="Permalink to this heading"></a></h3>
<section id="cephadm-paused">
<h4>CEPHADM_PAUSED<a class="headerlink" href="#cephadm-paused" title="Permalink to this heading"></a></h4>
<p>This indicates that cephadm background work has been paused with
<code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">pause</span></code>.  Cephadm continues to perform passive monitoring
activities (like checking host and daemon status), but it will not
make any changes (like deploying or removing daemons).</p>
<p>Resume cephadm work by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>resume</span>
</pre></div></div></section>
<section id="cephadm-stray-host">
<span id="id2"></span><h4>CEPHADM_STRAY_HOST<a class="headerlink" href="#cephadm-stray-host" title="Permalink to this heading"></a></h4>
<p>This indicates that one or more hosts have Ceph daemons that are
running, but are not registered as hosts managed by <em>cephadm</em>.  This
means that those services cannot currently be managed by cephadm
(e.g., restarted, upgraded, included in <cite>ceph orch ps</cite>).</p>
<ul>
<li><p>You can manage the host(s) by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>host<span class="w"> </span>add<span class="w"> </span>*&lt;hostname&gt;*</span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>You might need to configure SSH access to the remote host
before this will work.</p>
</div>
</li>
<li><p>See <a class="reference internal" href="../host-management/#cephadm-fqdn"><span class="std std-ref">Fully qualified domain names vs bare host names</span></a> for more information about host names and
domain names.</p></li>
<li><p>Alternatively, you can manually connect to the host and ensure that
services on that host are removed or migrated to a host that is
managed by <em>cephadm</em>.</p></li>
<li><p>This warning can be disabled entirely by running the following
command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/warn_on_stray_hosts<span class="w"> </span><span class="nb">false</span></span>
</pre></div></div></li>
</ul>
</section>
<section id="cephadm-stray-daemon">
<h4>CEPHADM_STRAY_DAEMON<a class="headerlink" href="#cephadm-stray-daemon" title="Permalink to this heading"></a></h4>
<p>One or more Ceph daemons are running but not are not managed by
<em>cephadm</em>.  This may be because they were deployed using a different
tool, or because they were started manually.  Those
services cannot currently be managed by cephadm (e.g., restarted,
upgraded, or included in <cite>ceph orch ps</cite>).</p>
<ul>
<li><p>If the daemon is a stateful one (monitor or OSD), it should be adopted
by cephadm; see <a class="reference internal" href="../adoption/#cephadm-adoption"><span class="std std-ref">现有集群切换到 cephadm</span></a>.  For stateless daemons, it is
usually easiest to provision a new daemon with the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span></code>
command and then stop the unmanaged daemon.</p></li>
<li><p>If the stray daemon(s) are running on hosts not managed by cephadm, you can manage the host(s) by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>host<span class="w"> </span>add<span class="w"> </span>*&lt;hostname&gt;*</span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>You might need to configure SSH access to the remote host
before this will work.</p>
</div>
</li>
<li><p>See <a class="reference internal" href="../host-management/#cephadm-fqdn"><span class="std std-ref">Fully qualified domain names vs bare host names</span></a> for more information about host names and
domain names.</p></li>
<li><p>This warning can be disabled entirely by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/warn_on_stray_daemons<span class="w"> </span><span class="nb">false</span></span>
</pre></div></div></li>
</ul>
</section>
<section id="cephadm-host-check-failed">
<h4>CEPHADM_HOST_CHECK_FAILED<a class="headerlink" href="#cephadm-host-check-failed" title="Permalink to this heading"></a></h4>
<p>One or more hosts have failed the basic cephadm host check, which verifies
that (1) the host is reachable and cephadm can be executed there, and (2)
that the host satisfies basic prerequisites, like a working container
runtime (podman or docker) and working time synchronization.
If this test fails, cephadm will not be able to manage services on that host.</p>
<p>You can manually run this check by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>cephadm<span class="w"> </span>check-host<span class="w"> </span>*&lt;hostname&gt;*</span>
</pre></div></div><p>You can remove a broken host from management by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>host<span class="w"> </span>rm<span class="w"> </span>*&lt;hostname&gt;*</span>
</pre></div></div><p>You can disable this health warning by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/warn_on_failed_host_check<span class="w"> </span><span class="nb">false</span></span>
</pre></div></div></section>
</section>
<section id="cluster-configuration-checks">
<h3>Cluster Configuration Checks<a class="headerlink" href="#cluster-configuration-checks" title="Permalink to this heading"></a></h3>
<p>Cephadm periodically scans each host in the cluster in order
to understand the state of the OS, disks, network interfacess etc. This information can
then be analyzed for consistency across the hosts in the cluster to
identify any configuration anomalies.</p>
<section id="enabling-cluster-configuration-checks">
<h4>Enabling Cluster Configuration Checks<a class="headerlink" href="#enabling-cluster-configuration-checks" title="Permalink to this heading"></a></h4>
<p>These configuration checks are an <strong>optional</strong> feature, and are enabled
by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/config_checks_enabled<span class="w"> </span><span class="nb">true</span></span>
</pre></div></div></section>
<section id="states-returned-by-cluster-configuration-checks">
<h4>States Returned by Cluster Configuration Checks<a class="headerlink" href="#states-returned-by-cluster-configuration-checks" title="Permalink to this heading"></a></h4>
<p>Configuration checks are triggered after each host scan. The
cephadm log entries will show the current state and outcome of the
configuration checks as follows:</p>
<p>Disabled state (config_checks_enabled false):</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>ALL<span class="w"> </span>cephadm<span class="w"> </span>checks<span class="w"> </span>are<span class="w"> </span>disabled,<span class="w"> </span>use<span class="w"> </span><span class="s1">&#39;ceph config set mgr mgr/cephadm/config_checks_enabled true&#39;</span><span class="w"> </span>to<span class="w"> </span><span class="nb">enable</span>
</pre></div>
</div>
<p>Enabled state (config_checks_enabled true):</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>CEPHADM<span class="w"> </span><span class="m">8</span>/8<span class="w"> </span>checks<span class="w"> </span>enabled<span class="w"> </span>and<span class="w"> </span>executed<span class="w"> </span><span class="o">(</span><span class="m">0</span><span class="w"> </span>bypassed,<span class="w"> </span><span class="m">0</span><span class="w"> </span>disabled<span class="o">)</span>.<span class="w"> </span>No<span class="w"> </span>issues<span class="w"> </span>detected
</pre></div>
</div>
</section>
<section id="managing-configuration-checks-subcommands">
<h4>Managing Configuration Checks (subcommands)<a class="headerlink" href="#managing-configuration-checks-subcommands" title="Permalink to this heading"></a></h4>
<p>The configuration checks themselves are managed through several cephadm subcommands.</p>
<p>To determine whether the configuration checks are enabled, run the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>cephadm<span class="w"> </span>config-check<span class="w"> </span>status</span>
</pre></div></div><p>This command returns the status of the configuration checker as either “Enabled” or “Disabled”.</p>
<p>To list all the configuration checks and their current states, run the following command:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp"># </span>ceph<span class="w"> </span>cephadm<span class="w"> </span>config-check<span class="w"> </span>ls

<span class="go">  NAME             HEALTHCHECK                      STATUS   DESCRIPTION</span>
<span class="go">kernel_security  CEPHADM_CHECK_KERNEL_LSM         enabled  check that SELINUX/Apparmor profiles are consistent across cluster hosts</span>
<span class="go">os_subscription  CEPHADM_CHECK_SUBSCRIPTION       enabled  check that subscription states are consistent for all cluster hosts</span>
<span class="go">public_network   CEPHADM_CHECK_PUBLIC_MEMBERSHIP  enabled  check that all hosts have a network interface on the Ceph public_network</span>
<span class="go">osd_mtu_size     CEPHADM_CHECK_MTU                enabled  check that OSD hosts share a common MTU setting</span>
<span class="go">osd_linkspeed    CEPHADM_CHECK_LINKSPEED          enabled  check that OSD hosts share a common network link speed</span>
<span class="go">network_missing  CEPHADM_CHECK_NETWORK_MISSING    enabled  check that the cluster/public networks as defined exist on the Ceph hosts</span>
<span class="go">ceph_release     CEPHADM_CHECK_CEPH_RELEASE       enabled  check for Ceph version consistency: all Ceph daemons should be the same release unless upgrade is in progress</span>
<span class="go">kernel_version   CEPHADM_CHECK_KERNEL_VERSION     enabled  checks that the maj.min version of the kernel is consistent across Ceph hosts</span>
</pre></div>
</div>
<p>The name of each configuration check can be used to enable or disable a specific check by running a command of the following form:
:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>cephadm<span class="w"> </span>config-check<span class="w"> </span>disable<span class="w"> </span>&lt;name&gt;</span>
</pre></div></div><p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>cephadm<span class="w"> </span>config-check<span class="w"> </span>disable<span class="w"> </span>kernel_security</span>
</pre></div></div></section>
<section id="cephadm-check-kernel-lsm">
<h4>CEPHADM_CHECK_KERNEL_LSM<a class="headerlink" href="#cephadm-check-kernel-lsm" title="Permalink to this heading"></a></h4>
<p>Each host within the cluster is expected to operate within the same Linux
Security Module (LSM) state. For example, if the majority of the hosts are
running with SELINUX in enforcing mode, any host not running in this mode is
flagged as an anomaly and a healthcheck (WARNING) state raised.</p>
</section>
<section id="cephadm-check-subscription">
<h4>CEPHADM_CHECK_SUBSCRIPTION<a class="headerlink" href="#cephadm-check-subscription" title="Permalink to this heading"></a></h4>
<p>This check relates to the status of OS vendor subscription. This check is
performed only for hosts using RHEL and helps to confirm that all hosts are
covered by an active subscription, which ensures that patches and updates are
available.</p>
</section>
<section id="cephadm-check-public-membership">
<h4>CEPHADM_CHECK_PUBLIC_MEMBERSHIP<a class="headerlink" href="#cephadm-check-public-membership" title="Permalink to this heading"></a></h4>
<p>All members of the cluster should have a network interface configured on at least one of the
public network subnets. Hosts that are not on the public network will rely on
routing, which may affect performance.</p>
</section>
<section id="cephadm-check-mtu">
<h4>CEPHADM_CHECK_MTU<a class="headerlink" href="#cephadm-check-mtu" title="Permalink to this heading"></a></h4>
<p>The MTU of the network interfaces on OSD hosts can be a key factor in consistent performance. This
check examines hosts that are running OSD services to ensure that the MTU is
configured consistently within the cluster. This is determined by determining
the MTU setting that the majority of hosts is using. Any anomalies result in a
health check.</p>
</section>
<section id="cephadm-check-linkspeed">
<h4>CEPHADM_CHECK_LINKSPEED<a class="headerlink" href="#cephadm-check-linkspeed" title="Permalink to this heading"></a></h4>
<p>This check is similar to the MTU check. Link speed consistency is a factor in
consistent cluster performance, as is the MTU of the OSD node network interfaces.
This check determines the link speed shared by the majority of OSD hosts, and a
health check is run for any hosts that are set at a lower link speed rate.</p>
</section>
<section id="cephadm-check-network-missing">
<h4>CEPHADM_CHECK_NETWORK_MISSING<a class="headerlink" href="#cephadm-check-network-missing" title="Permalink to this heading"></a></h4>
<p>The <cite>public_network</cite> and <cite>cluster_network</cite> settings support subnet definitions
for IPv4 and IPv6. If these settings are not found on any host in the cluster,
a health check is raised.</p>
</section>
<section id="cephadm-check-ceph-release">
<h4>CEPHADM_CHECK_CEPH_RELEASE<a class="headerlink" href="#cephadm-check-ceph-release" title="Permalink to this heading"></a></h4>
<p>Under normal operations, the Ceph cluster runs daemons that are of the same Ceph
release (for example, Reef).  This check determines the active release for each daemon, and
reports any anomalies as a healthcheck. <em>This check is bypassed if an upgrade
is in process.</em></p>
</section>
<section id="cephadm-check-kernel-version">
<h4>CEPHADM_CHECK_KERNEL_VERSION<a class="headerlink" href="#cephadm-check-kernel-version" title="Permalink to this heading"></a></h4>
<p>The OS kernel version (maj.min) is checked for consistency across hosts.
The kernel version of the majority of the hosts is used as the basis for
identifying anomalies.</p>
</section>
</section>
</section>
<section id="client-keyrings-and-configs">
<span id="id3"></span><h2>Client keyrings and configs<a class="headerlink" href="#client-keyrings-and-configs" title="Permalink to this heading"></a></h2>
<p>Cephadm can distribute copies of the <code class="docutils literal notranslate"><span class="pre">ceph.conf</span></code> file and client keyring
files to hosts. Starting from versions 16.2.10 (Pacific) and 17.2.1 (Quincy),
in addition to the default location <code class="docutils literal notranslate"><span class="pre">/etc/ceph/</span></code> cephadm also stores config
and keyring files in the <code class="docutils literal notranslate"><span class="pre">/var/lib/ceph/&lt;fsid&gt;/config</span></code> directory. It is usually
a good idea to store a copy of the config and <code class="docutils literal notranslate"><span class="pre">client.admin</span></code> keyring on any host
used to administer the cluster via the CLI. By default, cephadm does this for any
nodes that have the <code class="docutils literal notranslate"><span class="pre">_admin</span></code> label (which normally includes the bootstrap host).</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Ceph daemons will still use files on <code class="docutils literal notranslate"><span class="pre">/etc/ceph/</span></code>. The new configuration
location <code class="docutils literal notranslate"><span class="pre">/var/lib/ceph/&lt;fsid&gt;/config</span></code> is used by cephadm only. Having this config
directory under the fsid helps cephadm to load the configuration associated with
the cluster.</p>
</div>
<p>When a client keyring is placed under management, cephadm will:</p>
<blockquote>
<div><ul class="simple">
<li><p>build a list of target hosts based on the specified placement spec (see
<a class="reference internal" href="../services/#orchestrator-cli-placement-spec"><span class="std std-ref">Daemon Placement</span></a>)</p></li>
<li><p>store a copy of the <code class="docutils literal notranslate"><span class="pre">/etc/ceph/ceph.conf</span></code> file on the specified host(s)</p></li>
<li><p>store a copy of the <code class="docutils literal notranslate"><span class="pre">ceph.conf</span></code> file at <code class="docutils literal notranslate"><span class="pre">/var/lib/ceph/&lt;fsid&gt;/config/ceph.conf</span></code> on the specified host(s)</p></li>
<li><p>store a copy of the <code class="docutils literal notranslate"><span class="pre">ceph.client.admin.keyring</span></code> file at <code class="docutils literal notranslate"><span class="pre">/var/lib/ceph/&lt;fsid&gt;/config/ceph.client.admin.keyring</span></code> on the specified host(s)</p></li>
<li><p>store a copy of the keyring file on the specified host(s)</p></li>
<li><p>update the <code class="docutils literal notranslate"><span class="pre">ceph.conf</span></code> file as needed (e.g., due to a change in the cluster monitors)</p></li>
<li><p>update the keyring file if the entity’s key is changed (e.g., via <code class="docutils literal notranslate"><span class="pre">ceph</span>
<span class="pre">auth</span> <span class="pre">...</span></code> commands)</p></li>
<li><p>ensure that the keyring file has the specified ownership and specified mode</p></li>
<li><p>remove the keyring file when client keyring management is disabled</p></li>
<li><p>remove the keyring file from old hosts if the keyring placement spec is
updated (as needed)</p></li>
</ul>
</div></blockquote>
<section id="listing-client-keyrings">
<h3>Listing Client Keyrings<a class="headerlink" href="#listing-client-keyrings" title="Permalink to this heading"></a></h3>
<p>To see the list of client keyrings are currently under management, run the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>client-keyring<span class="w"> </span>ls</span>
</pre></div></div></section>
<section id="putting-a-keyring-under-management">
<h3>Putting a Keyring Under Management<a class="headerlink" href="#putting-a-keyring-under-management" title="Permalink to this heading"></a></h3>
<p>To put a keyring under management, run a command of the following form:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>client-keyring<span class="w"> </span><span class="nb">set</span><span class="w"> </span>&lt;entity&gt;<span class="w"> </span>&lt;placement&gt;<span class="w"> </span><span class="o">[</span>--mode<span class="o">=</span>&lt;mode&gt;<span class="o">]</span><span class="w"> </span><span class="o">[</span>--owner<span class="o">=</span>&lt;uid&gt;.&lt;gid&gt;<span class="o">]</span><span class="w"> </span><span class="o">[</span>--path<span class="o">=</span>&lt;path&gt;<span class="o">]</span></span>
</pre></div></div><ul class="simple">
<li><p>By default, the <em>path</em> is <code class="docutils literal notranslate"><span class="pre">/etc/ceph/client.{entity}.keyring</span></code>, which is
where Ceph looks by default.  Be careful when specifying alternate locations,
as existing files may be overwritten.</p></li>
<li><p>A placement of <code class="docutils literal notranslate"><span class="pre">*</span></code> (all hosts) is common.</p></li>
<li><p>The mode defaults to <code class="docutils literal notranslate"><span class="pre">0600</span></code> and ownership to <code class="docutils literal notranslate"><span class="pre">0:0</span></code> (user root, group root).</p></li>
</ul>
<p>For example, to create a <code class="docutils literal notranslate"><span class="pre">client.rbd</span></code> key and deploy it to hosts with the
<code class="docutils literal notranslate"><span class="pre">rbd-client</span></code> label and make it group readable by uid/gid 107 (qemu), run the
following commands:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>auth<span class="w"> </span>get-or-create-key<span class="w"> </span>client.rbd<span class="w"> </span>mon<span class="w"> </span><span class="s1">&#39;profile rbd&#39;</span><span class="w"> </span>mgr<span class="w"> </span><span class="s1">&#39;profile rbd&#39;</span><span class="w"> </span>osd<span class="w"> </span><span class="s1">&#39;profile rbd pool=my_rbd_pool&#39;</span></span>
<span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>client-keyring<span class="w"> </span><span class="nb">set</span><span class="w"> </span>client.rbd<span class="w"> </span>label:rbd-client<span class="w"> </span>--owner<span class="w"> </span><span class="m">107</span>:107<span class="w"> </span>--mode<span class="w"> </span><span class="m">640</span></span>
</pre></div></div><p>The resulting keyring file is:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">-rw-r-----. 1 qemu qemu 156 Apr 21 08:47 /etc/ceph/client.client.rbd.keyring</span>
</pre></div>
</div>
<p>By default, cephadm will also manage <code class="docutils literal notranslate"><span class="pre">/etc/ceph/ceph.conf</span></code> on hosts where it writes the keyrings.
This feature can be suppressed by passing <code class="docutils literal notranslate"><span class="pre">--no-ceph-conf</span></code> when setting the keyring.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>client-keyring<span class="w"> </span><span class="nb">set</span><span class="w"> </span>client.foo<span class="w"> </span>label:foo<span class="w"> </span><span class="m">0</span>:0<span class="w"> </span>--no-ceph-conf</span>
</pre></div></div></section>
<section id="disabling-management-of-a-keyring-file">
<h3>Disabling Management of a Keyring File<a class="headerlink" href="#disabling-management-of-a-keyring-file" title="Permalink to this heading"></a></h3>
<p>To disable management of a keyring file, run a command of the following form:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>client-keyring<span class="w"> </span>rm<span class="w"> </span>&lt;entity&gt;</span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>This deletes any keyring files for this entity that were previously written
to cluster nodes.</p>
</div>
</section>
</section>
<section id="etc-ceph-ceph-conf">
<span id="etc-ceph-conf-distribution"></span><h2>/etc/ceph/ceph.conf<a class="headerlink" href="#etc-ceph-ceph-conf" title="Permalink to this heading"></a></h2>
<section id="distributing-ceph-conf-to-hosts-that-have-no-keyrings">
<h3>Distributing ceph.conf to hosts that have no keyrings<a class="headerlink" href="#distributing-ceph-conf-to-hosts-that-have-no-keyrings" title="Permalink to this heading"></a></h3>
<p>It might be useful to distribute <code class="docutils literal notranslate"><span class="pre">ceph.conf</span></code> files to hosts without an
associated client keyring file.  By default, cephadm deploys only a
<code class="docutils literal notranslate"><span class="pre">ceph.conf</span></code> file to hosts where a client keyring is also distributed (see
above).  To write config files to hosts without client keyrings, run the
following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/manage_etc_ceph_ceph_conf<span class="w"> </span><span class="nb">true</span></span>
</pre></div></div></section>
<section id="using-placement-specs-to-specify-which-hosts-get-keyrings">
<h3>Using Placement Specs to specify which hosts get keyrings<a class="headerlink" href="#using-placement-specs-to-specify-which-hosts-get-keyrings" title="Permalink to this heading"></a></h3>
<p>By default, the configs are written to all hosts (i.e., those listed by <code class="docutils literal notranslate"><span class="pre">ceph</span>
<span class="pre">orch</span> <span class="pre">host</span> <span class="pre">ls</span></code>).  To specify which hosts get a <code class="docutils literal notranslate"><span class="pre">ceph.conf</span></code>, run a command of
the following form:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/manage_etc_ceph_ceph_conf_hosts<span class="w"> </span>&lt;placement<span class="w"> </span>spec&gt;</span>
</pre></div></div><p>For example, to distribute configs to hosts with the <code class="docutils literal notranslate"><span class="pre">bare_config</span></code> label, run
the following command:</p>
</section>
<section id="distributing-ceph-conf-to-hosts-tagged-with-bare-config">
<h3>Distributing ceph.conf to hosts tagged with bare_config<a class="headerlink" href="#distributing-ceph-conf-to-hosts-tagged-with-bare-config" title="Permalink to this heading"></a></h3>
<p>For example, to distribute configs to hosts with the <code class="docutils literal notranslate"><span class="pre">bare_config</span></code> label, run the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/manage_etc_ceph_ceph_conf_hosts<span class="w"> </span>label:bare_config</span>
</pre></div></div><p>(See <a class="reference internal" href="../services/#orchestrator-cli-placement-spec"><span class="std std-ref">Daemon Placement</span></a> for more information about placement specs.)</p>
</section>
</section>
<section id="limiting-password-less-sudo-access">
<h2>Limiting Password-less sudo Access<a class="headerlink" href="#limiting-password-less-sudo-access" title="Permalink to this heading"></a></h2>
<p>By default, the cephadm install guide recommends enabling password-less
<code class="docutils literal notranslate"><span class="pre">sudo</span></code> for the cephadm user. This option is the most flexible and
future-proof but may not be preferred in all environments. An administrator can
restrict <code class="docutils literal notranslate"><span class="pre">sudo</span></code> to only running an exact list of commands without password
access.  Note that this list may change between Ceph versions and
administrators choosing this option should read the release notes and review
this list in the destination version of the Ceph documentation. If the list
differs one must extend the list of password-less <code class="docutils literal notranslate"><span class="pre">sudo</span></code> commands prior to
upgrade.</p>
<p>Commands requiring password-less sudo support:</p>
<blockquote>
<div><ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">chmod</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">chown</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ls</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">mkdir</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">mv</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">rm</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">sysctl</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">touch</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">true</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">which</span></code> (see note)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">/usr/bin/cephadm</span></code> or python executable (see note)</p></li>
</ul>
</div></blockquote>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Typically cephadm will execute <code class="docutils literal notranslate"><span class="pre">which</span></code> to determine what python3
command is available and then use the command returned by <code class="docutils literal notranslate"><span class="pre">which</span></code> in
subsequent commands.
Before configuring <code class="docutils literal notranslate"><span class="pre">sudo</span></code> run <code class="docutils literal notranslate"><span class="pre">which</span> <span class="pre">python3</span></code> to determine what
python command to add to the <code class="docutils literal notranslate"><span class="pre">sudo</span></code> configuration.
In some rare configurations <code class="docutils literal notranslate"><span class="pre">/usr/bin/cephadm</span></code> will be used instead.</p>
</div>
<p>Configuring the <code class="docutils literal notranslate"><span class="pre">sudoers</span></code> file can be performed using a tool like <code class="docutils literal notranslate"><span class="pre">visudo</span></code>
and adding or replacing a user configuration line such as the following:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># assuming the cephadm user is named &quot;ceph&quot;</span>
<span class="n">ceph</span> <span class="n">ALL</span><span class="o">=</span><span class="p">(</span><span class="n">ALL</span><span class="p">)</span> <span class="n">NOPASSWD</span><span class="p">:</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">chmod</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">chown</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">ls</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">mkdir</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">mv</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">rm</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">sbin</span><span class="o">/</span><span class="n">sysctl</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">touch</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">true</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">which</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">cephadm</span><span class="p">,</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">python3</span>
</pre></div>
</div>
</section>
<section id="purging-a-cluster">
<h2>Purging a cluster<a class="headerlink" href="#purging-a-cluster" title="Permalink to this heading"></a></h2>
<div class="admonition danger">
<p class="admonition-title">Danger</p>
<p>THIS OPERATION WILL DESTROY ALL DATA STORED IN THIS CLUSTER</p>
</div>
<p>In order to destroy a cluster and delete all data stored in this cluster, disable
cephadm to stop all orchestration operations (so we avoid deploying new daemons).</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>mgr<span class="w"> </span>module<span class="w"> </span>disable<span class="w"> </span>cephadm</span>
</pre></div></div><p>Then verify the FSID of the cluster:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>fsid</span>
</pre></div></div><p>Purge ceph daemons from all hosts in the cluster</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="c1"># For each host:</span></span>
<span class="prompt1">cephadm<span class="w"> </span>rm-cluster<span class="w"> </span>--force<span class="w"> </span>--zap-osds<span class="w"> </span>--fsid<span class="w"> </span>&lt;fsid&gt;</span>
</pre></div></div></section>
<section id="replacing-a-device">
<h2>Replacing a device<a class="headerlink" href="#replacing-a-device" title="Permalink to this heading"></a></h2>
<p>The <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">device</span> <span class="pre">replace</span></code> command automates the process of replacing the underlying device of an OSD.
Previously, this process required manual intervention at various stages.
With this new command, all necessary operations are performed automatically, streamlining the replacement process
and improving the overall user experience.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This only supports LVM-based deployed OSD(s)</p>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>device<span class="w"> </span>replace<span class="w"> </span>&lt;host&gt;<span class="w"> </span>&lt;device-path&gt;</span>
</pre></div></div><p>In the case the device being replaced is shared by multiple OSDs (eg: DB/WAL device shared by multiple OSDs), the orchestrator will warn you.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="o">[</span>ceph:<span class="w"> </span>root@ceph<span class="w"> </span>/<span class="o">]</span><span class="c1"># ceph orch device replace osd-1 /dev/vdd</span></span>
<span class="prompt1"></span>
<span class="prompt1">Error<span class="w"> </span>EINVAL:<span class="w"> </span>/dev/vdd<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span>shared<span class="w"> </span>device.</span>
<span class="prompt1">Replacing<span class="w"> </span>/dev/vdd<span class="w"> </span>implies<span class="w"> </span>destroying<span class="w"> </span>OSD<span class="o">(</span>s<span class="o">)</span>:<span class="w"> </span><span class="o">[</span><span class="s1">&#39;0&#39;</span>,<span class="w"> </span><span class="s1">&#39;1&#39;</span><span class="o">]</span>.</span>
<span class="prompt1">Please,<span class="w"> </span>*be<span class="w"> </span>very<span class="w"> </span>careful*,<span class="w"> </span>this<span class="w"> </span>can<span class="w"> </span>be<span class="w"> </span>a<span class="w"> </span>very<span class="w"> </span>dangerous<span class="w"> </span>operation.</span>
<span class="prompt1">If<span class="w"> </span>you<span class="w"> </span>know<span class="w"> </span>what<span class="w"> </span>you<span class="w"> </span>are<span class="w"> </span>doing,<span class="w"> </span>pass<span class="w"> </span>--yes-i-really-mean-it</span>
</pre></div></div><p>If you know what you are doing, you can go ahead and pass <code class="docutils literal notranslate"><span class="pre">--yes-i-really-mean-it</span></code>.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="o">[</span>ceph:<span class="w"> </span>root@ceph<span class="w"> </span>/<span class="o">]</span><span class="c1"># ceph orch device replace osd-1 /dev/vdd --yes-i-really-mean-it</span></span>
<span class="prompt1"><span class="w">  </span>Scheduled<span class="w"> </span>to<span class="w"> </span>destroy<span class="w"> </span>osds:<span class="w"> </span><span class="o">[</span><span class="s1">&#39;6&#39;</span>,<span class="w"> </span><span class="s1">&#39;7&#39;</span>,<span class="w"> </span><span class="s1">&#39;8&#39;</span><span class="o">]</span><span class="w"> </span>and<span class="w"> </span>mark<span class="w"> </span>/dev/vdd<span class="w"> </span>as<span class="w"> </span>being<span class="w"> </span>replaced.</span>
</pre></div></div><p><code class="docutils literal notranslate"><span class="pre">cephadm</span></code> will make <code class="docutils literal notranslate"><span class="pre">ceph-volume</span></code> zap and destroy all related devices and mark the corresponding OSD as <code class="docutils literal notranslate"><span class="pre">destroyed</span></code> so the
different OSD(s) ID(s) will be preserved:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="o">[</span>ceph:<span class="w"> </span>root@ceph-1<span class="w"> </span>/<span class="o">]</span><span class="c1"># ceph osd tree</span></span>
<span class="prompt1"><span class="w">  </span>ID<span class="w">  </span>CLASS<span class="w">  </span>WEIGHT<span class="w">   </span>TYPE<span class="w"> </span>NAME<span class="w">         </span>STATUS<span class="w">     </span>REWEIGHT<span class="w">  </span>PRI-AFF</span>
<span class="prompt1"><span class="w">  </span>-1<span class="w">         </span><span class="m">0</span>.97659<span class="w">  </span>root<span class="w"> </span>default</span>
<span class="prompt1"><span class="w">  </span>-3<span class="w">         </span><span class="m">0</span>.97659<span class="w">      </span>host<span class="w"> </span>devel-1</span>
<span class="prompt1"><span class="w">   </span><span class="m">0</span><span class="w">    </span>hdd<span class="w">  </span><span class="m">0</span>.29300<span class="w">          </span>osd.0<span class="w">     </span>destroyed<span class="w">   </span><span class="m">1</span>.00000<span class="w">  </span><span class="m">1</span>.00000</span>
<span class="prompt1"><span class="w">   </span><span class="m">1</span><span class="w">    </span>hdd<span class="w">  </span><span class="m">0</span>.29300<span class="w">          </span>osd.1<span class="w">     </span>destroyed<span class="w">   </span><span class="m">1</span>.00000<span class="w">  </span><span class="m">1</span>.00000</span>
<span class="prompt1"><span class="w">   </span><span class="m">2</span><span class="w">    </span>hdd<span class="w">  </span><span class="m">0</span>.19530<span class="w">          </span>osd.2<span class="w">            </span>up<span class="w">   </span><span class="m">1</span>.00000<span class="w">  </span><span class="m">1</span>.00000</span>
<span class="prompt1"><span class="w">   </span><span class="m">3</span><span class="w">    </span>hdd<span class="w">  </span><span class="m">0</span>.19530<span class="w">          </span>osd.3<span class="w">            </span>up<span class="w">   </span><span class="m">1</span>.00000<span class="w">  </span><span class="m">1</span>.00000</span>
</pre></div></div><p>The device being replaced is finally seen as <code class="docutils literal notranslate"><span class="pre">being</span> <span class="pre">replaced</span></code> preventing <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> from redeploying the OSDs too fast:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="o">[</span>ceph:<span class="w"> </span>root@ceph-1<span class="w"> </span>/<span class="o">]</span><span class="c1"># ceph orch device ls</span></span>
<span class="prompt1">HOST<span class="w">     </span>PATH<span class="w">      </span>TYPE<span class="w">  </span>DEVICE<span class="w"> </span>ID<span class="w">   </span>SIZE<span class="w">  </span>AVAILABLE<span class="w">  </span>REFRESHED<span class="w">  </span>REJECT<span class="w"> </span>REASONS</span>
<span class="prompt1">osd-1<span class="w">  </span>/dev/vdb<span class="w">  </span>hdd<span class="w">               </span>200G<span class="w">  </span>Yes<span class="w">        </span>13s<span class="w"> </span>ago</span>
<span class="prompt1">osd-1<span class="w">  </span>/dev/vdc<span class="w">  </span>hdd<span class="w">               </span>200G<span class="w">  </span>Yes<span class="w">        </span>13s<span class="w"> </span>ago</span>
<span class="prompt1">osd-1<span class="w">  </span>/dev/vdd<span class="w">  </span>hdd<span class="w">               </span>200G<span class="w">  </span>Yes<span class="w">        </span>13s<span class="w"> </span>ago<span class="w">    </span>Is<span class="w"> </span>being<span class="w"> </span>replaced</span>
<span class="prompt1">osd-1<span class="w">  </span>/dev/vde<span class="w">  </span>hdd<span class="w">               </span>200G<span class="w">  </span>No<span class="w">         </span>13s<span class="w"> </span>ago<span class="w">    </span>Has<span class="w"> </span>a<span class="w"> </span>FileSystem,<span class="w"> </span>Insufficient<span class="w"> </span>space<span class="w"> </span><span class="o">(</span>&lt;<span class="m">10</span><span class="w"> </span>extents<span class="o">)</span><span class="w"> </span>on<span class="w"> </span>vgs,<span class="w"> </span>LVM<span class="w"> </span>detected</span>
<span class="prompt1">osd-1<span class="w">  </span>/dev/vdf<span class="w">  </span>hdd<span class="w">               </span>200G<span class="w">  </span>No<span class="w">         </span>13s<span class="w"> </span>ago<span class="w">    </span>Has<span class="w"> </span>a<span class="w"> </span>FileSystem,<span class="w"> </span>Insufficient<span class="w"> </span>space<span class="w"> </span><span class="o">(</span>&lt;<span class="m">10</span><span class="w"> </span>extents<span class="o">)</span><span class="w"> </span>on<span class="w"> </span>vgs,<span class="w"> </span>LVM<span class="w"> </span>detected</span>
</pre></div></div><p>If for any reason you need to clear the ‘device replace header’ on a device, then you can use <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">device</span> <span class="pre">replace</span> <span class="pre">&lt;host&gt;</span> <span class="pre">&lt;device&gt;</span> <span class="pre">--clear</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="o">[</span>ceph:<span class="w"> </span>root@devel-1<span class="w"> </span>/<span class="o">]</span><span class="c1"># ceph orch device replace devel-1 /dev/vdk --clear</span></span>
<span class="prompt1">Replacement<span class="w"> </span>header<span class="w"> </span>cleared<span class="w"> </span>on<span class="w"> </span>/dev/vdk</span>
<span class="prompt1"><span class="o">[</span>ceph:<span class="w"> </span>root@devel-1<span class="w"> </span>/<span class="o">]</span><span class="c1">#</span></span>
</pre></div></div><p>After that, <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> will redeploy the OSD service spec within a few minutes (unless the service is set to <code class="docutils literal notranslate"><span class="pre">unmanaged</span></code>).</p>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../upgrade/" class="btn btn-neutral float-left" title="升级 Ceph" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../client-setup/" class="btn btn-neutral float-right" title="Basic Ceph Client Setup" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>